Search-Based Mutant Selection for Efficient Test Suite Improvement: Evaluation and Results

Context: Search-based techniques have been applied to almost all areas in software engineering, especially to software testing, seeking to solve hard optimization problems. However, the problem of selecting mutants to improve the test suite at a lower cost has not been explored to the same extent as other problems, such as mutant selection for test suite evaluation or test data generation. Objective: In this paper, we apply search-based mutant selection to enhance the quality of test suites efficiently. Namely, we use the technique known as Evolutionary Mutation Testing (EMT), which allows reducing the number of mutants while preserving the power to refine the test suite. Despite reported benefits of its application, the existing empirical results were derived from a limited number of case studies, a particular set of mutation operators and a vague measure, which currently makes it difficult to determine the real performance of this technique. Method: This paper addresses the shortcomings of previous studies, providing a new methodology to evaluate EMT on the basis of the actual improvement of the test suite achieved by using the evolutionary strategy. We make use of that methodology in new experiments with a carefully selected set of real-world C++ case studies. Results: EMT shows a good performance for most case studies and levels of demand of test suite improvement (around 45% less mutants than random selection in the best case). The results reveal that even a reduced subset of mutants selected with EMT can serve to increase confidence in the test suite, especially in programs with a large set of mutants. Conclusions: These results support the use of search-based techniques to solve the problem of mutant selection for a more efficient test suite refinement. Additionally, we identify some aspects that could foreseeably help enhance EMT

Proceedings of the Doctoral Consortium in Computer Science (JIPII 2021)

Actas de las Jornadas de Investigación Predoctoral en Ingeniería InformáticaThis volume contains the proceedings of the Primeras Jornadas de Investigación Predoctoral en Ingeniería Informática - First Doctoral Consortium in Computer Science, JIPII 2021, which was held online on June 15th, 2021. The aim of JIPII 2021 was to provide a forum for PhD students to present and discuss their research under the guidance of a panel of senior researchers. The advances in their PhD theses under development in the Doctoral Program in Computer Science were presented in the Consortium. This Doctoral Program belongs to the Doctoral School of the University of Cadiz (EDUCA). Different stages of research were covered, from the most incipient phase, such as the PhD Thesis plans (or even a Master’s Thesis), to the most advanced phases in which the defence of the PhD Thesis is imminent. We enjoyed twenty very nice and interesting talks, organized in four sessions. We had a total of fifty participants, including speakers and attendees, with an average of thirty-two people in the morning sessions and an average of twenty people in the afternoon sessions. Several people contributed to the success of JIPII 2021. We are grateful to the Academic Committee of the Doctoral Program in Computer Science and the School of Engineering for their support. We would like also to thank the Program Committee for their work in reviewing the papers, as well as all the students and supervisors for their interest and participation. Finally, the proceedings have been published by the Department of Computer Science and Engineering. We hope that you find the proceedings useful, interesting, and challenging

MEdit4CEP: A model-driven solution for real-time decision making in SOA 2.0

Organizations all around the world need to manage huge amounts of data from heterogeneous sources every day in order to conduct decision making processes. This requires them to infer what the value of such data is for the business in question through data analysis as well as acting promptly for critical or relevant situations. Complex Event Processing (CEP) is a technology that helps tackle this issue by detecting event patterns in real time. However, this technology forces domain experts to define these patterns indicating such situations and the appropriate actions to be executed in their information systems, generally based on Service-Oriented Architectures (SOAs). In particular, these users face the incommodity of implementing these patterns manually or by using editors which are not user-friendly enough. To deal with this problem, a model-driven solution for real-time decision making in event-driven SOAs is proposed and conducted in this paper. This approach allows the integration of CEP with this architecture type as well as defining CEP domain and event pattern through a graphical and intuitive editor, which also permits automatic code generation. Moreover, the solution is evaluated and its benefits are discussed. As a result, we can assert this is a novel solution for bringing CEP technology closer to any user, positively impacting on business decision making processes

Using Evolutionary Mutation Testing to improve the quality of test suites

Mutation testing is a method used to assess and improve the fault detection capability of a test suite by creating faulty versions, called mutants, of the system under test. Evolutionary Mutation Testing (EMT), like selective mutation or mutant sampling, was proposed to reduce the computational cost, which is a major concern when applying mutation testing. This technique implements an evolutionary algorithm to produce a reduced subset of mutants but with a high proportion of mutants that can help the tester derive new test cases (strong mutants). In this paper, we go a step further in estimating the ability of this technique to induce the generation of test cases. Instead of measuring the percentage of strong mutants within the subset of generated mutants, we compute how much the test suite is actually improved thanks to those mutants. In our experiments, we have compared the extent to which EMT and the random selection of mutants help to find missing test cases in C++ object-oriented systems. We can conclude from our results that the percentage of mutants generated with EMT is lower than with the random strategy to obtain a test suite of the same size and that the technique scales better for complex programs

ModeL4CEP: Graphical domain-specific modeling languages for CEP domains and event patterns

Complex event processing (CEP) is a cutting-edge technology that allows the analysis and correlation of large volumes of data with the aim of detecting complex and meaningful events through the use of event patterns, as well as permitting the inference of valuable knowledge for end users. Despite the great advantages that CEP can bring to expert or intelligent business systems, it poses a substantial challenge to their users, who are business experts but do not have the necessary knowledge and experience using this technology. The main problem these users have to face is precisely hand-writing the code for event pattern definition, which requires them to implement the conditions to be met to detect relevant situations for the domain in question by using a particular event processing language (EPL). In order to respond to this need, in this paper we propose both a graphical domain-specific modeling language (DSML) for facilitating CEP domain definitions by domain experts, and a graphical DSML for event pattern definition by non-technological users. The proposed languages provide high expressiveness and flexibility and are independent of event patterns and actions’ implementation code. This way, domain experts can define the relevant event types and patterns within their business domain, without having to be experts on EPL programming, nor on other complicated computer science technological issues, beyond an understandable and intuitive graphical definition. Furthermore, with these DSMLs, users will also be able to define the actions to be automatically taken once a pattern is detected in the system. Further benefits of these DSMLs are evaluated and discussed in depth in this paper

IoT-TEG 4.0: A New Approach 4.0 for Test Event Generation

The Industry 4.0 (I4.0) is a paradigm settled down by the introduction of the Internet of things (IoT) into the production and manufacturing environment. I4.0 promotes the connection of physical items such as sensors, devices, and enterprise assets, to each other and to the Internet. The information that flows through these items is vital because it serves to make relevant decisions. One of the main features of I4.0 is its adaptability to the human needs, this means that the items included in the I4.0 network are heterogeneous and they are large in number. The majority of I4.0 papers, which are focused on testing, describe a specific system or part of the I4.0 network. We have not found any paper that undertakes the testing of multiple connected IoT devices that will receive, process, and make decisions according to the complex and real data that travel through the network. In this article, we present IoT-TEG (Test Event Generator) 4.0, which is based on the test event generator system IoT-TEG . IoT-TEG 4.0 provides two new main contributions: the generation of test cases, which can include all the different types of data that the connected I4.0 devices under study can manage, and real-time testing. Additionally, its validation using real IoT programs is included and the results show that IoT-TEG 4.0 allows us to conduct tests that mimic real IoT system behaviors

Un algoritmo híbrido para el problema NRP con interdependencias

En este artículo presentamos un algoritmo híbrido para una variante del problema de la siguiente versión (NRP). En esta variante existe un conjunto de requisitos para los que se dispone de una estimación del esfuerzo necesario para su implementación y de la satisfacción percibida por los potenciales clientes con la inclusión de dichos requisitos. Entre estos requisitos existen relaciones de interdependencia, que establecen a ciertos requisitos como prerequisitos de otros, o que obligan a implementar determinados requisitos simultáneamente en caso de incluirse alguno de ellos en la siguiente versión del producto a desarrollar. Dado un límite superior de esfuerzo prefijado, el objetivo es seleccionar un subconjunto de requisitos que cumpla todas las restricciones y maximice la satisfacción global de los clientes. La propuesta combina heurísticas con técnicas exactas para una versión simplificada del problema. El rendimiento del algoritmo resultante en distintos escenarios realistas se compara con el de otras técnicas metaheurísticas previamente empleada

Performance mutation testing: Hypothesis and open questions

Performance bugs are common, costly, and elusive. Performance tests aim to detect performance bugs by running the program with specific inputs and determining whether the observed behaviour is acceptable. There not exist mechanisms, however, to assess the effectiveness of performance tests. Mutation testing is a technique to evaluate and enhance functional test suites by seeding artificial faults in the program under test. In this new idea paper, we explore the applicability of mutation testing to assess and improve performance tests. This novel approach is motivated with examples and open questions

Performance mutation testing

Performance bugs are known to be a major threat to the success of software products. Performance tests aim to detect performance bugs by executing the program through test cases and checking whether it exhibits a noticeable performance degradation. The principles of mutation testing, a well-established testing technique for the assessment of test suites through the injection of artificial faults, could be exploited to evaluate and improve the detection power of performance tests. However, the application of mutation testing to assess performance tests, henceforth called performance mutation testing (PMT), is a novel research topic with numerous open challenges. In previous papers, we identified some key challenges related to PMT. In this work, we go a step further and explore the feasibility of applying PMT at the source-code level in general-purpose languages. To do so, we revisit concepts associated with classical mutation testing, and design seven novel mutation operators to model known bug-inducing patterns. As a proof of concept, we applied traditional mutation operators as well as performance mutation operators to open-source C++ programs. The results reveal the potential of the new performance-mutants to help assess and enhance performance tests when compared to traditional mutants. A review of live mutants in these programs suggests that they can induce the design of special test inputs. In addition to these promising results, our work brings a whole new set of challenges related to PMT, which will hopefully serve as a starting point for new contributions in the area

TANDEM: A Taxonomy and a Dataset of Real-World Performance Bugs

The detection of performance bugs, like those causing an unexpected execution time, has gained much attention in the last years due to their potential impact in safety-critical and resource-constrained applications. Much effort has been put on trying to understand the nature of performance bugs in different domains as a starting point for the development of effective testing techniques. However, the lack of a widely accepted classification scheme of performance faults and, more importantly, the lack of well-documented and understandable datasets makes it difficult to draw rigorous and verifiable conclusions widely accepted by the community. In this paper, we present TANDEM, a dual contribution related to real-world performance bugs. Firstly, we propose a taxonomy of performance bugs based on a thorough systematic review of the related literature, divided into three main categories: effects, causes and contexts of bugs. Secondly, we provide a complete collection of fully documented real-world performance bugs. Together, these contributions pave the way for the development of stronger and reproducible research results on performance testing